logo
down
shadow

Trying to concat two time series dataframes and matchup the timestamps as close as possible


Trying to concat two time series dataframes and matchup the timestamps as close as possible

By : Sam
Date : October 17 2020, 01:08 AM
wish of those help How can I concatenate two pandas dataframes with a datetime index, so that the timestamps match up as close as possible. Could there be a padding method to use? , Not sure if this would do but if you use reindex before concat,
code :
df2 = df2.reindex(df1.index)
df3 = pd.concat([df1.reset_index().add_suffix('_x'),\ 
df2.reset_index().add_suffix('_y')], axis=1)\
.set_index(['index_x', 'index_y']).sort_index(axis=1)


Share : facebook icon twitter icon
Aggregate multiple time series in Mongodb, where all time series have the same timestamps available

Aggregate multiple time series in Mongodb, where all time series have the same timestamps available


By : mikesearch
Date : March 29 2020, 07:55 AM
I wish this helpful for you I have multiple time series in database in mongodb, with fields "ticker", "time", and "close" amongst other fields: , See if the following is in line with what you want to accomplish:
code :
db.bbticks.aggregate(
[
 { $match: { time: { $gte: ISODate("2013-08-01") } } },
 { $group: { _id: "$time", count: {$sum: 1}, tickers: { $push: { "ticker": "$ticker" , "close": "$close" } } } } ,
 { $match: { count: { $gt: 1 } } }
]
)
var mapFunction = function() {
                      var key = this.time

                      var value = { tickers: [
                                                { ticker: this.ticker, close: this.close } 
                                             ] };

                      emit( key, value );
                  };

var reduceFunction = function(keyObject, valuesArray) {
                     var reducedValue = { tickers: [] };

                     for (var idx = 0; idx < valuesArray.length; idx++) {
                        reducedValue.tickers.push( valuesArray[idx].tickers[0] )
                     }

                     return reducedValue;
                  };


db.bbticks.mapReduce( mapFunction,
                      reduceFunction,
                      {
                        out: "mr_interim_results",
                        sort: { time: 1 },
                        query: {
                                 time: {$gte: ISODate("2013-08-01") }
                               },
                      }
                   )

db.mr_interim_results.find( { 'value.tickers': { $not: { $size: 1 } } }  )
How do I turn this series of dataframes into a time series in Pandas?

How do I turn this series of dataframes into a time series in Pandas?


By : yaganteeswarudu
Date : March 29 2020, 07:55 AM
wish help you to fix your issue If you have a list of data frames, you can provide a key as the dates while doing the concatenation and then unstack the original index to columns:
code :
pd.concat([df1, df2], keys=['1/1/2013', '1/2/2013']).unstack(level=1)
concat sliced dataframes preserving original series order

concat sliced dataframes preserving original series order


By : VijayBagada
Date : March 29 2020, 07:55 AM
fixed the issue. Will look into that further I think not, need sort_index for sorting columns names:
code :
df = pd.concat(frames, axis = 1).sort_index(axis=1)
L = list1 + list2 + list3
df1 = pd.concat(frames, axis = 1).reindex(columns=df.columns[sorted(L)])
df_1 = df.iloc[:,sorted(list1)]
#carry out preprocessing
df_2 = df.iloc[:,sorted(list2)]
#carry out preprocessing
df_3 = df.iloc[:,sorted(list3)]
#carry out preprocessing
np.random.seed(100)
df = pd.DataFrame(np.random.randint(10, size=(5,10)), columns=list('EFGHIJABCD'))
print (df)
   E  F  G  H  I  J  A  B  C  D
0  8  8  3  7  7  0  4  2  5  2
1  2  2  1  0  8  4  0  9  6  2
2  4  1  5  3  4  4  3  7  1  1
3  7  7  0  2  9  9  3  2  5  8
4  1  0  7  6  2  0  8  2  5  1

list1 = [1,4,3]
list2 = [2,5,6]
list3 = [9,8,7]
df_1 = df.iloc[:,list1]
#carry out preprocessing
df_2 = df.iloc[:,list2]
#carry out preprocessing
df_3 = df.iloc[:,list3]
#carry out preprocessing

#join the frames back together
frames = [df_1,df_2,df_3]
L = list1 + list2 + list3

df1 = pd.concat(frames, axis = 1).reindex(columns=df.columns[sorted(L)])
print (df1)
   F  G  H  I  J  A  B  C  D
0  8  3  7  7  0  4  2  5  2
1  2  1  0  8  4  0  9  6  2
2  1  5  3  4  4  3  7  1  1
3  7  0  2  9  9  3  2  5  8
4  0  7  6  2  0  8  2  5  1

df2 = pd.concat(frames, axis = 1).sort_index(axis=1)
print (df2)
   A  B  C  D  F  G  H  I  J
0  4  2  5  2  8  3  7  7  0
1  0  9  6  2  2  1  0  8  4
2  3  7  1  1  1  5  3  4  4
3  3  2  5  8  7  0  2  9  9
4  8  2  5  1  0  7  6  2  0
L.sort()
df = df[L]
df = df[sorted(L)]
Pandas concat similar DataFrames and Series

Pandas concat similar DataFrames and Series


By : user2347188
Date : March 29 2020, 07:55 AM
will be helpful for those in need I have a list of Dataframes, all with the same columns. Occaisionally, a DataFrame has only one row, and is, hence, a Series. When I try to concatenate this list with pd.concat, where there was a Series, it puts what I want to be the columns in the index. See below for a minimal working example.
code :
thing2 = pd.DataFrame(thing2).transpose()
pd.concat([thing1, thing2, thing3])
Concat dataframes/series with axis=1 in a loop

Concat dataframes/series with axis=1 in a loop


By : user3191453
Date : March 29 2020, 07:55 AM
Any of those help Consider pivot_table after calculating month_end (see @Root's answer). Also, use reindex to fill in missing months. Usually in Pandas, grouping aggregations like count of senders per month does not require looping or temporary helper data frames.
code :
from pandas.tseries.offsets import MonthEnd

df_in['month_end'] = (df_in['time'] + MonthEnd(0)).dt.normalize()

agg_df = (df_in.pivot_table(index='month_end', columns='sender', values='time', aggfunc='count')
               .reindex(pd.date_range('1998-01-01', '2000-01-31', freq='m').values, axis='index')
               .fillna(0)                
          )
print(agg_df)  
# sender      Able Boy  Mark L. Taylor  james h. madison  james joyce  scott kirk
# month_end                                                                      
# 1998-01-31       0.0             0.0               0.0          0.0         0.0
# 1998-02-28       0.0             0.0               0.0          0.0         0.0
# 1998-03-31       0.0             0.0               0.0          0.0         0.0
# 1998-04-30       0.0             0.0               0.0          0.0         0.0
# 1998-05-31       0.0             0.0               0.0          0.0         0.0
# 1998-06-30       0.0             0.0               0.0          0.0         0.0
# 1998-07-31       0.0             0.0               0.0          0.0         0.0
# 1998-08-31       0.0             0.0               0.0          0.0         0.0
# 1998-09-30       0.0             0.0               0.0          0.0         0.0
# 1998-10-31       0.0             0.0               0.0          0.0         0.0
# 1998-11-30       4.0             3.0               0.0          0.0         0.0
# 1998-12-31       1.0             0.0               0.0          0.0         4.0
# 1999-01-31       1.0             0.0               4.0          0.0         0.0
# 1999-02-28       0.0             0.0               0.0          0.0         0.0
# 1999-03-31       0.0             0.0               0.0          0.0         0.0
# 1999-04-30       0.0             0.0               0.0          0.0         0.0
# 1999-05-31       0.0             0.0               0.0          0.0         0.0
# 1999-06-30       0.0             0.0               0.0          0.0         0.0
# 1999-07-31       0.0             0.0               0.0          0.0         0.0
# 1999-08-31       0.0             0.0               0.0          0.0         0.0
# 1999-09-30       0.0             0.0               0.0          0.0         0.0
# 1999-10-31       0.0             0.0               0.0          0.0         0.0
# 1999-11-30       0.0             0.0               0.0          0.0         0.0
# 1999-12-31       0.0             0.0               0.0          0.0         0.0
# 2000-01-31       0.0             0.0               0.0          4.0         1.0
shadow
Privacy Policy - Terms - Contact Us © soohba.com