Creating a generator function for a large data set for a neural network The Next CEO of Stack OverflowSetting MaxIterations for neural network function `Predict`?Neural Network for polynomial fitImplementing a Neural NetworkHow to make similar objects have similar embedding weights?Neural network giving different answersImage Generator with a Neural Network (CIFAR-10)Backpropagation for manual neural network trainingMonitoring the Training Progress through PutAppendCreating an indicator function in a neural networkMatlab Neural Network Toolbox vs Wolfram Neural Network Framework
Trouble understanding the speech of overseas colleagues
Why does C# sound extremely flat when saxophone is tuned to G?
What makes a siege story/plot interesting?
Why does GHC infer a monomorphic type here, even with MonomorphismRestriction disabled?
Should I tutor a student who I know has cheated on their homework?
Unreliable Magic - Is it worth it?
Return of the Riley Riddles in Reverse
What happens if you roll doubles 3 times then land on "Go to jail?"
Can a single photon have an energy density?
Why were Madagascar and New Zealand discovered so late?
Why here is plural "We went to the movies last night."
How to write papers efficiently when English isn't my first language?
How to Reset Passwords on Multiple Websites Easily?
What can we do to stop prior company from asking us questions?
Why do remote companies require working in the US?
How do scammers retract money, while you can’t?
What does this shorthand mean?
How can I open an app using Terminal?
Can the Reverse Gravity spell affect the Meteor Swarm spell?
How do I construct this japanese bowl?
Inappropriate reference requests from Journal reviewers
Example of a Mathematician/Physicist whose Other Publications during their PhD eclipsed their PhD Thesis
When airplanes disconnect from a tanker during air to air refueling, why do they bank so sharply to the right?
Why does standard notation not preserve intervals (visually)
Creating a generator function for a large data set for a neural network
The Next CEO of Stack OverflowSetting MaxIterations for neural network function `Predict`?Neural Network for polynomial fitImplementing a Neural NetworkHow to make similar objects have similar embedding weights?Neural network giving different answersImage Generator with a Neural Network (CIFAR-10)Backpropagation for manual neural network trainingMonitoring the Training Progress through PutAppendCreating an indicator function in a neural networkMatlab Neural Network Toolbox vs Wolfram Neural Network Framework
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited 11 hours ago
m_goldberg
87.9k872198
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
$endgroup$
add a comment |
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited 11 hours ago
m_goldberg
87.9k872198
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
$endgroup$
add a comment |
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited 11 hours ago
m_goldberg
87.9k872198
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
$endgroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
neural-networks
edited 11 hours ago
m_goldberg
87.9k872198
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
edited 11 hours ago
m_goldberg
87.9k872198
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
edited 11 hours ago
m_goldberg
87.9k872198
edited 11 hours ago
m_goldberg
87.9k872198
edited 11 hours ago
m_goldberg
87.9k872198
87.9k872198
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
asked 12 hours ago
Michel MesedahlMichel Mesedahl
262
262
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
11 hours ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f194098%2fcreating-a-generator-function-for-a-large-data-set-for-a-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
11 hours ago
add a comment |
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
11 hours ago
add a comment |
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
answered 11 hours ago
Alexey GolyshevAlexey Golyshev
5,0971739
5,0971739
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
11 hours ago
add a comment |
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
11 hours ago
$begingroup$
Note: Mathematica seems to ignore
RoundLength for my generator. Play with MaxTrainingRounds.$endgroup$
– Alexey Golyshev
11 hours ago
$begingroup$
Note: Mathematica seems to ignore
RoundLength for my generator. Play with MaxTrainingRounds.$endgroup$
– Alexey Golyshev
11 hours ago
add a comment |
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f194098%2fcreating-a-generator-function-for-a-large-data-set-for-a-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
